Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add mask as optional input to UltimateSD Sampler #96

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

Naomi-Ken-Korem
Copy link

No description provided.

@ssitu
Copy link
Owner

ssitu commented Jul 24, 2024

I have questions about this. Is it intended that the mask is applied to each chunk of the image? If it is intended, how useful is it? I can't think of a good way to use this.

@Naomi-Ken-Korem
Copy link
Author

You can add one mask that is applied on all frames in video, or a batch of masks ( mask for each frame, which makes more sense this is how I use it)
It is a way to get results like here:
https://discord.com/channels/1076117621407223829/1264492978232229908

In practice it enables to apply the animatediff model using ultimateupscale only on part on the areas in the video. ( and preserve the consistency of the masked area )

@ssitu
Copy link
Owner

ssitu commented Jul 24, 2024

Looks like I need access to that discord channel if I want to see the attachment.

@Naomi-Ken-Korem
Copy link
Author

it is the banadoco channel https://discord.gg/pn385fGy
Here it is :)

@ssitu
Copy link
Owner

ssitu commented Jul 28, 2024

So if I'm getting this right, the mask can be used to avoid rediffusing an area where you want to prioritize consistency while also upscaling the frames? Although I'm still confused about how adding a mask to ultimate upscale would help. The mask will be used in each chunk of the upscale, which can lead to strange results. Here is a demonstration with 4 tiles, the blurred areas are the parts where the resampling happens:

image

Wouldn't you want the mask to take up the entire upscaled image/frame instead of each tile?

@Naomi-Ken-Korem
Copy link
Author

Yes, you are right, I mostly worked with the redundant setup where the tiles are same size os the full image, and than there is no reason to actually use UltimateSD

I will try to fix the pr

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants